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The Science-in-CTE pilot study tested a curriculum integration model that enhanced the science that oc¬ 
curs in CTE curricula. The study replicated the National Research Center for Career and Technical Ed¬ 
ucation’s (NRCCTE) Math-in-CTE experimental research design (Stone, Alfeld, & Pearson, 2008) with 
applied science in secondary agricultural education. The semester-length study was conducted in North 
Dakota with secondary agricultural educators who were randomly selected to participate in the experi¬ 
mental and control groups. The experimental treatment mirrored the Math-in-CTE model of extended 
professional development, partnering experimental CTE group teachers with science educators, and use 
of a 7-element pedagogic framework. Standardized measures of science achievemen t were administered 
to students to determine the impact of the treatment on their science knowledge and skills. The results of 
hierarchical linear modeling (HEM) analysis indicated that the intervention had a statistically significant 
positive impact on posttest science achievement for students in the 2 nd , 3 rd , and 4 th qucirtiles on pretest 
science achievement. We interpreted the intervention’s effects as small for participants in the 2 nd quartile 
and as moderate for those in the 3 rd and 4' h qucirtiles. Relative to the control group, the intervention ap¬ 
peared to have no impact on students in the 1 st quartile on pretest science achievement. 
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The Science-in-CTE pilot study tested a 
model of curriculum integration that enhances 
the instruction of science in career and technical 
education (CTE) curricula. The pilot replicated 
the experimental research design of the Math-in- 
CTE study conducted by the National Research 
Center for Career and Technical Education 
(NRCCTE) from 2004-2005 (Stone, Alfeld, 
Pearson, Lewis, & Jensen, 2006). Findings from 
the Math-in-CTE study provided evidence that 
enhancing the instruction of mathematics in 
CTE curricula could improve students’ math 
skills. Students in experimental/treatment class¬ 
rooms scored significantly higher on standard¬ 


ized tests of achievement in mathematics than 
did their counterparts in control classrooms. 
Furthermore, students’ academic improvements 
came without loss of occupational knowledge 
and skills (Stone et al., 2008). The success of 
the Math-in-CTE study generated additional 
testable questions regarding other possibilities 
for academic integration. Specifically, 
NRCCTE researchers considered whether it 
would be possible to adapt the model to inte¬ 
grate science with CTE. 

The decline of national science achievement 
scores added urgency to this question. Con¬ 
sistent with the trends in math achievement that 
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led to the Math-in-CTE study, National Assess¬ 
ment of Educational Progress (NAEP, 2005) 
data on science achievement showed that student 
scores had not increased in spite of an increase 
of required science credits from 1.4 in the mid- 
1980s to 3.2 in 2004 (Silverberg, Warner, Fong, 
& Goodwin, 2004). In fact, NAEP data showed 
that at the Grade 12 level, the average score for 
science achievement had declined since 1996. 
In 2005, only 54% of students scored at or above 
the Basic level on the science exam (NAEP, 
2005). Collectively, such data not only chal¬ 
lenged the prevailing assumption that requiring 
more science credits would lead to increased 
knowledge and skills, but also suggested the 
need for a different approach to science educa¬ 
tion. 

CTE and Academic Curriculum Integration 

The content of career and technical educa¬ 
tion is driven by the needs of the workplace; 
therefore CTE instructors maintain a close con¬ 
nection with “real work.” Examination of CTE 
curricula through mapping and other kinds of 
integration strategies has revealed a context rich 
with authentic opportunities for strengthening 
academic skills (Zirkle, 2004). The Math-in- 
CTE study, upon which the Science-in-CTE 
study described in this paper was designed, 
clearly demonstrated that CTE teachers can 
identify the significant amount of academic 
knowledge and skills that are embedded in the 
technical content, while providing their academ¬ 
ic counterparts with real-life, problem-based 
activities through which students can apply aca¬ 
demics in relevant ways (Stone et al., 2008). 

When the Science-in-CTE study was first 
proposed, few studies were found that addressed 
the impact of CTE-science integration on stu¬ 
dent achievement, although the field of CTE has 
had a long-term priority focus on curriculum 
integration (Hoachlander, 1999). An earlier 
study conducted by Roegge and Russell (1990) 
utilized a quasi-experimental design to test the 
effect of incorporating biology principles into a 
unit of instruction in vocational agriculture on 
student achievement and attitudes. Roegge and 
Russell found the integrated approach superior 
to the traditional approach in producing higher 
overall achievement and higher applied biology 


achievement. They also reported that students 
had a more positive attitude toward the learning 
experience. Similarly, Conroy and Walker 
(2000) conducted interviews with teachers and 
students that revealed that students performed 
better in math and science classes as a result of 
what they learned in aquaculture. These early 
studies suggested the need and potential for fur¬ 
ther research on the integration of CTE and sci¬ 
ence. 

The integration of academics into the CTE 
curriculum was a major policy objective of the 
Carl D. Perkins Career and Technical Education 
Improvement Act of 2006, also known as Per¬ 
kins IV. The Perkins IV legislation explicitly 
linked professional development (PD) to im¬ 
proved teaching practices and student outcomes 
and required that PD in career and technical 
fields promote “the integration of coherent and 
rigorous academic content standards and career 
and technical education curricula, including ... 
opportunities for the appropriate academic and 
career and technical education teachers to jointly 
develop and implement curricula and pedagogi¬ 
cal standards.” Furthermore, the PD was to be 
“high quality, sustained, and intensive” and was 
to increase academic knowledge (Carl D. Per¬ 
kins Career and Technical Education Improve¬ 
ment Act of 2006, S. 250, Sec. 122 (c) (A), p. 
36). Converging with the stagnation of science 
achievement and the proven success of mathe¬ 
matics integration, this legislative mandate pro¬ 
vided further justification for testing a CTE- 
science integration model. 

Science Instruction in Context 

Concurrent with the Perkins IV legislative 
mandate was an emerging conversation among 
science professionals about traditional science 
teaching methods—specifically, the need to pro¬ 
vide a relevant context through which to learn 
science concepts and principles. Gilbert (2006) 
summarized the discussion in this way: 

(a) Because of high content loads, the sci¬ 
ence curricula are too often aggregations 
of isolated facts detached from their sci¬ 
entific origin; 

(b) Students do not know how they should 
connect the aggregations of isolated 
facts that do not lend themselves to the 
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formation of coherent mental schema 
and give no meaning to what they have 
learned; 

(c) Students fail to solve problems using the 
same concepts in other situations than 
those that closely mirror the ways in 
which they were taught. 

(d) Students do not feel a sense of why they 
learn the material required; it does not 
become relevant for them. 

(e) The traditional emphases of the science 
curriculum (“solid foundation,” “correct 
explanation,” and “scientific skill devel¬ 
opment”) are increasingly seen as an in¬ 
adequate basis for the more advanced 
study of science, (p. 958; emphasis in 
the original) 

Bennett, Lubben, and Hogarth (2007) offered 
this definition of a context-based approach to 
science education: 

Context-based approaches are ap¬ 
proaches adopted in science teaching 
where contexts and applications of sci¬ 
ence are used as the starting point for 
the development of scientific ideas. 
This contrasts with more traditional ap¬ 
proaches that cover scientific ideas first, 
before looking at applications, (p. 248; 
emphasis in the original) 

These authors reported that context-based sci¬ 
ence courses motivate students and help them 
feel more positive about science by helping them 
see the importance of what they are studying. 
When students are more interested and motivat¬ 
ed by the experiences they are having in their 
lessons, their increased engagement may result 
in improved learning (Bennet et al., 2007). 

Munby, Taylor, Chin, and Hutchinson 
(2007) defined differences between school sci¬ 
ence and workplace science in terms of purpose, 
emphasizing that students in science-rich work¬ 
places frequently do not make connections be¬ 
tween their work tasks and the academic science 
taught in school. The authors suggested that this 
occurs in part because workplace science often 
masks the underlying science by transforming 
science-based procedures into routine sequences. 
Munby et al. (2007) further suggested that meta- 
cognitive instruction could be helpful for stu¬ 
dents to both (a) understand their work tasks 
more fully by understanding the underlying sci¬ 


ence and (b) simultaneously improve their fun¬ 
damental knowledge of academic science. 

The National Science Education Standards 
published by the National Committee on Science 
Education Standards and Assessment and the 
National Research Council (2007) provided a 
number of guidelines for improving K-12 sci¬ 
ence education. Among these methods were: (a) 
learning through inquiry, (b) “hands-on” learn¬ 
ing, (c) “minds-on” learning, and (d) “science as 
process.” (p. 2). These guidelines recommend 
that teachers begin with the problems that arise 
at work or in the community and introduce the 
scientific ideas needed to identify and test possi¬ 
ble solutions to those problems as opposed to 
beginning with abstract principles of science and 
adding application as an afterthought. 

An emerging model supporting knowledge 
integration was found among those in the sci¬ 
ence community who argued for “coherent sci¬ 
ence education” (Kali, Linn, & Roseman, 2008). 
In doing so, they promoted a movement beyond 
standards to a more systematic approach to sci¬ 
ence instruction that makes explicit the connec¬ 
tions among scientific concepts and principles. 
The authors further promoted contextualized 
science learning through the use of real-world 
problems and inquiry-based projects. Specifi¬ 
cally, the Center for Curriculum Materials in 
Science (CCMS) approach emphasized curricu¬ 
lar coherence based on these characteristics: (a) 
“interconnectedness of core knowledge,” (b) 
“connections between ideas of science and phe¬ 
nomena in the real world,” (c) “co nn ections be¬ 
tween new ideas and prior knowledge,” and (d) 
“connections between scientific ideas and the 
enterprise that produced them” (Roseman, Linn, 
& Koppal, 2008, pp. 16-25). The CCMS model¬ 
ing, based on empirical research, closely paral¬ 
leled the approach of the evidence-based contex¬ 
tualized approach of the Math-in-CTE model, 
which proved to increase students’ math skills. 

Theoretical Underpinnings 

The theoretical underpinnings of a contextu¬ 
alized approach can be traced back to Dewey 
(1916), who proposed that students should be 
educated through occupations rather than for 
occupations. “Rather than conceptualizing nar¬ 
row, specific job skills as the goal of occupa- 
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tional courses, Dewey argued that occupational 
contexts could provide a rich venue in which 
students could effectively learn important fun¬ 
damental concepts in traditional subject matter” 
(NRCCTE Curriculum Integration Workgroup, 

2010, p. 2). 

Situated Cognition 

As with the Math-in-CTE study, the Sci- 
ence-in-CTE pilot study drew heavily upon the 
theory of situated cognition (Lave & Wenger, 
1991). Situated cognition is a complex interplay 
between physical and social context, authenticity 
of experience, and personal construction of 
knowledge (Darvin, 2006; Hendricks, 2001). 
Although it is sometimes difficult to fine-tune 
the terminology, most agree that it represents a 
“move away from the individual as an isolated 
unit into which knowledge may or may not be 
‘plugged’” (Moore, 1998, p. 170). As the 
NRCCTE Curriculum Integration Workgroup 
(2010) reported: 

Numerous theories exist as to how expe¬ 
riential learning works and for whom, 
educators generally agree on the value 
of relevance and application. The im¬ 
portance of relevance is prominent in 
the experiential learning theory of situ¬ 
ated cognition, which promotes the con¬ 
text in which learning occurs as being 
central to understanding (Brown, Col¬ 
lins, & Duguid, 1989; Paige & Daley, 
2009). Brown and colleagues (1989) 
suggested that situated cognition re¬ 
quires us to consider knowledge as a set 
of tools that can (a) be useful only when 
understood and (b) vary in purpose 
based on the situation at hand. Situated 
cognition emphasizes learning experi¬ 
ences in “authentic versus decontextual- 
ized contexts” (Choi & Hannafm, 1995, 
as cited in Merriam et al., 2007, p. 180). 
(P- 7) 

The contextualized approach used in the Sci- 
ence-in-CTE model required that CTE teachers 
change their way of teaching to more actively 
engage students in learning science and in think¬ 
ing as scientists (National Research Council, 
2000). As with the Math-in-CTE model, the 
authentic context of CTE served to situate sci¬ 


ence in ways that helped students cognitively 
make sense of the academic content (Brown et 
al., 1989; Cognition and Technology Group at 
Vanderbilt, 1990; as cited in Stone et al., 2008). 

Conceptual Framework 

A major activity of the Science-in-CTE pilot 
study was to develop the conceptual framework 
that would guide the integration process. The 
research team, with input from a science educa¬ 
tion consultant, drafted a framework that was 
subsequently presented to an advisory group of 
science and CTE educators for feedback and 
further refinement. 

The overall approach to the CTE and sci¬ 
ence integration reflected the Math-in-CTE 
model in which the CTE concepts within the 
curriculum dictated the selection and use of the 
academic content. Therefore, as a first step, the 
research team adopted the Math-in-CTE peda¬ 
gogic (instructional) framework to ensure that 
the transfer of learning elements were retained in 
the development of the science-enhanced CTE 
lessons (Stone et al., 2008). These seven ele¬ 
ments, as below, were used to guide teachers’ 
development of the integrated lessons and their 
subsequent instruction: 

1. Introduce the CTE lesson. 

2. Assess students’ science awareness as 
it relates to the CTE lesson. 

3. Work through the science embedded 
in the CTE lesson. 

4. Work through related, contextual sci- 
ence-in-CTE examples. 

5. Work through explicit science exam¬ 
ples. 

6. Students demonstrate their understand¬ 
ing. 

7. Formal assessment. (Adapted from the 
Math-in-CTE model, Stone et al., 
2008) 

This process of solving a real, relevant problem, 
practicing several similar examples, and then 
applying the concept to a more abstract problem 
(transfer of learning, Elements 3-5 of the seven- 
element model) was also congruent with the 
characteristics of connectedness called for in 
coherent science instruction (Kali et al., 2008). 

For the purpose of the pilot study, and with 
input from an advisory committee, the research 
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team used the term explicit science to express 
the embedded science as it might appear in a 
science classroom or on a standardized test. 
Drawing from the science education literature, 
the research team surmised that the science ex¬ 
plicit in CTE curricula would represent any of 
three overlapping domains of science knowledge 


and skills (see Figure 1). These domains, from 
which teachers could draw to create integrated 
lessons, included science concepts and unifying 
principles, the nature of science (NOS), and sci¬ 
entific inquiry (Biological Sciences Curriculum 
Study [BSCS], 2006; Flick, 2006; Schwartz & 
Crawford, 2006; Scott, Asoko, & Feach, 2007). 
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Figure 1. The relationship of science education perspectives to the pedagogic framework. 


Notably, this conceptual model reflected a 
movement in the science education community 
away from the “process versus content” debate 
toward helping students understand science as it 
occurs in the natural world (BSCS, 2006). The 
research team hypothesized that the pedagogic 
framework developed for this pilot study would 
provide opportunities for students to learn sci¬ 
ence within the CTE context in ways that were 
complementary to these perspectives in the sci¬ 
ence education community. Guided by this 
framework, the CTE-science teacher teams par¬ 
ticipating in the pilot developed CTE lessons 
through which students not only learned the fac¬ 
tual information of science, but also engaged in 
the more complex interactions of scientific con¬ 
cepts and principles and scientific inquiry 


(BSCS, 2006; Flick, 2006; Schwartz & Craw¬ 
ford, 2006; Scott et al., 2007). 

Purpose and Research Question 

The purpose of the Science-in-CTE pilot 
study was to test a model of curriculum integra¬ 
tion that enhanced the science embedded in CTE 
curricula. The following research question 
guided the study: Is there a significant differ¬ 
ence in the scores of students who received sci¬ 
ence-enhanced CTE instruction and those of 
students in control classrooms, as measured by a 
standardized test of scientific knowledge? 
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Methods and Procedures 

The Science-in-CTE pilot study replicated 
the experimental design of the NRCCTE’s 
Math-in-CTE study (Stone et al., 2008), as 
adapted to test integrated science instruction. 
The intervention consisted of a process of ex¬ 
tended PD and a seven-element pedagogic 
framework that was used to develop and teach 
science-enhanced CTE lessons. Pre- and post¬ 
tests were administered to students in the class¬ 
rooms of participating teachers to determine the 
impact of the treatment on their science 
knowledge and skills. 

Experimental Design 

The semester-length pilot was conducted in 
Spring 2010 with CTE teachers in North Dakota. 
Replicating the research design used for the 
Math-in-CTE study, the research team utilized a 
group randomized trial (GRT; Jacob, Zhu, & 
Bloom, 2009; Murray, 1998). Twenty-nine ag¬ 
ricultural education teachers who volunteered to 
participate were randomly assigned to either an 
experimental group (15 teachers) or a control 
group (14 teachers). Random assignment of 
teachers from a common pool of those who vol¬ 
unteered was used to distribute any unmeasured 
factors that might be related to outcome varia¬ 
bles to the two conditions. Because analyses 
were conducted at the classroom level, a pretest 
of student science achievement was used to en¬ 
sure that the random assignment of teachers 
yielded comparable classrooms (Fraenkel & 
Wallen, 2003). 


Intervention 

The experimental group CTE teachers re¬ 
ceived six days of extended PD patterned after 
the model tested in the Math-in-CTE study. 
They were paired with science teacher partners 
and engaged with them in a community of prac¬ 
tice to map curricula, develop science-enhanced 
lessons, and critique and refine lessons (see Ta¬ 
ble 1). 

The Science-in-CTE model was not de¬ 
signed as a team-teaching model; rather, the ag¬ 
ricultural education teachers taught the enhanced 
CTE lessons on their own. They received ongo¬ 
ing support from their science teacher partners at 
their schools and participated in two days of ad¬ 
ditional PD at a midpoint in the pilot treatment 
period. During the PD sessions, each CTE- 
science teacher team created one science- 
enhanced lesson for a total of 15 for the group. 
Each of the lessons was presented before the full 
group; feedback was solicited and refinements 
were made. Each CTE teacher prepared an im¬ 
plementation calendar and committed to teach 
all 15 of the science-enhanced lessons during the 
treatment period. 

The agricultural education teachers random¬ 
ly assigned to the control groups were asked to 
conduct business-as-usual and to refrain from 
changing their instruction or curriculum during 
the treatment period. At the conclusion of the 
treatment period, the experimental teacher teams 
reconvened for a one-day session that included 
debriefing and focus groups; control teachers 
were invited to attend a one-day workshop at 
which they were debriefed and received training 
in the model and all instructional materials de¬ 
veloped by the experimental teachers. 
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Table 1 

Pilot Study Research Design and Activities by Semester 


Fall Semester 2009 Design and Activities Spring Semester 2010 Design and Activities 


Teacher groups 
randomly assigned 
in the GRT 

Initial PD 

Student 
Pretest 
and Survey 

Pilot Intervention 
Activities 

Student 

Posttest 

Experimental group: 

15 agricultural educa¬ 
tion teachers with sci¬ 
ence teacher partners 

4 days 

Yes 

Agricultural teachers 
teach science- 
enhanced lessons 
Agriculture-science 
teacher teams meet 
before each lesson 

2 days of PD at mid¬ 
semester 

Yes 

Control group: 14 ag¬ 
ricultural education 
teachers 

No PD 

Yes 

Conduct business-as- 
usual; 

No change to curricu¬ 
lum or instruction 

Yes 


Student Pretesting and Posttesting 

Students in classrooms of participating agri¬ 
cultural education experimental and control 
group teachers were pre- and posttested using 
standardized measures of science achievement. 
The selection of McGraw-Hill Terra Nova test 
batteries as appropriate instruments for this pur¬ 
pose was made by a science education specialist 
in conjunction with NRCCTE researchers. This 
selection was confirmed by a panel of experts 
who were convened at the beginning of the 
study to advise on the adaptation of the Math-in- 
CTE data collection instruments and pedagogic 
framework for use with science curricula. The 
science section of the Terra Nova Third Edition 
CAT Survey Level 21/22 Form C was used as 
the achievement pretest. The science section of 
the TerraNova, Third Edition, Complete Battery, 
Level 21/22 was utilized as the achievement 
posttest. 

Fidelity of Treatment Measures 

The research team employed multiple fideli¬ 
ty of treatment measures (a) to ensure that the 
enhanced lessons were implemented as planned 


and (b) to ascertain the extent to which they 
were instructed. These measures included: (a) 
monitoring of lesson calendars, (b) pre-teaching 
meeting reports submitted by the science teach¬ 
ers, (c) post-teaching reports submitted by CTE 
teachers, (d) collection of student work, and (e) 
debriefing of teachers in the focus groups. 

Analyses 

In this study, data for 376 secondary stu¬ 
dents were available for analysis. Due to coding 
errors, we could not determine classroom mem¬ 
bership for 13 participants and removed these 
cases. We handled all other missing data using 
the expectation maximization (EM) algorithm, 
which is available in the IBM SPSS 19 software 
package. The percentage of data missing for the 
variables ranged from 6.1% to 19.0%, with an 
average of 8.4%. A test of the assumption that 
the data were not missing completely at random 
(MCAR) yielded a non-significant result, sug¬ 
gesting that the imputation method was appro¬ 
priate (Little, 1988; Little & Schenker, 1995). 

Following these procedures, data for 363 
participants and 29 classrooms were available 
for analysis, with an average of 13 students per 
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classroom. Table 2 displays demographic in¬ 
formation for the cases before and after missing 
data were imputed. Due to the small number of 
ethnic minority students in the sample, we col¬ 
lapsed these participants into one minority group 

Table 2 


category. This group of 32 participants was 
comprised of 15 American Indian or Alaska Na¬ 
tives, 6 Asians, 1 Native Hawaiian or other Pa¬ 
cific Islander, and 9 participants who identified 
themselves as of more than one ethnicity. 


Sample Demographics 



Before Imputation 

After Imputation 


N 

Percent 

N 

Percent 

Condition 

Control 

164 

54.7 

192 

52.9 

Treatment 

136 

45.3 

171 

47.1 

Sex 

Male 

227 

69.4 

263 

72.5 

Female 

100 

30.6 

100 

27.5 

Ethnicity 

White 

290 

90.1 

331 

91.2 

Nonwhite 

32 

9.9 

32 

OO 
O 6 

Grade Level 

9 th grade 

104 

31.8 

104 

28.7 

10 th grade 

69 

21.1 

102 

28.1 

11 th grade 

68 

20.8 

71 

19.5 

12 th grade 

86 

26.3 

86 

23.7 


Hierarchical Linear Modeling 

We used hierarchical linear modeling 
(HLM) to test the effects of the intervention on 
science achievement, specifying a two-level 
model that incorporated both student- and class- 
room-level predictors into a single analysis. We 
chose HLM for three reasons. First, our testing 
data were from students nested in classrooms, 
and this methodology has been successfully used 
in the treatment of such data, where Type I error 
rates may be inflated by the similarity among 
members of a cluster (Raudenbush & Bryk, 
2002). Second, HLM allowed us to treat the 


intervention as a characteristic of classrooms 
and test its effects at Level 2, a more accurate 
statistical representation of our procedure. Fi¬ 
nally, this statistical methodology allowed us to 
control for the possibility that the classrooms 
differed on pretest science achievement by en¬ 
tering mean classroom pretest achievement as a 
Level 2 predictor of the outcome. We carried 
out the HLM analyses using full maximum like¬ 
lihood estimation and the HLM 6 software. See 
Table 3 for the predictors used in our analysis 
and see Table 4 for coding, centering, and equa¬ 
tions. 
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Table 3 


Descriptive Statistics for Level 1 and 2 Variables 



N 

Mean 

SD 

Min 

Max 



Level 1 Variables 


Posttest Number Correct 

363 

12.02 

6.49 

6.00 

39.00 

Age 

363 

16.04 

1.36 

12.00 

20.00 

Science Courses 

363 

2.17 

1.47 

0.00 

9.00 

Sex 

363 

0.69 

0.44 

0.00 

1.00 

Grade Level 

363 

1.40 

1.13 

0.00 

3.00 

Pretest Quartile 

363 

1.45 

1.12 

0.00 

3.00 

Minority 

363 

.10 

.28 

0.00 

1.00 



Level 2 Variables 


Class Mean Pretest 

29 

14.33 

1.99 

9.20 

17.50 

Group 

29 

.52 

.51 

0.00 

1.00 


Results 

For the null model, we observed that mean 
posttest scores for all models were statistically 
significantly different from zero {ps < .001). We 
also observed an intra-class correlation of . 13 for 
the null model, indicating that 13% of the vari¬ 
ance in science achievement was between class¬ 


rooms. Finally, we observed significant be- 
tween-group variance in the intercept for the null 
model {p < .001). We interpreted these results 
as suggesting that it was appropriate to use 
HLM. 


Table 4 
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Model Characteristics 


Predictor 


Description 


Coding (if needed) 


Student level predictors: 
Age a 

Science courses taken 3 
Sex 

Grade level 
Pretest quartile 


Minority 

Classroom level predictors: 
Group 

Class mean pretest score 1 


Chronological age 

Total number of science courses 

taken 

Biological sex 
Student academic level 

Quartile on pretest science 
achievement for each student 

Reported ethnic background col¬ 
lapsed into Caucasian and minor¬ 
ity 

Treatment condition for each 
classroom 

Mean pretest science score for 
each classroom 


0 = female and 1 = male 
0 = freshman, 1 = sophomore, 2 
= junior, and 3 = senior 
0 = 1 st quartile , 1 = 2 nd quartile, 
2 - 3 rd quartile, and 3 = 4 th 
quartile 

0 = Caucasian and 
1 = minority 

0 = control group and 1 = 
treatment group 


Equations: 

The Level 1 model, including the within class variables listed above: 

Posttest science achievement,, = p 0| + age,, + p 2 j science courses y + p 3j sex,, + 

P 4 j grade level y + p 5 j pretest quartile y + p 6 j minority y + r,, 


The Level 2 model, including the between class variables listed above: 

Poj = Yoo + Yoi group j + y 0 2 class mean pretest score, + u 0 ■, 

Pij = Yio + My 
Pzj = Y20 + «2j 
P3j = Y30 + “3j 
Pq = Y40 + «4j 

p5, = Y50 + Y5i group j + y 52 class mean pretest score, + u$, 

_ P6i ~ 760 + u 6] _ 

Note. 3 Predictors were grand mean centered. 

HLM Level 1 Model 

We specified and tested a Level 1 model to assess the degree to which student characteristics ex¬ 
plained the variance in mean posttest science achievement (see Table 5 for parameter estimates). Initially, 
we estimated random effects for all predictors. In the Level 1 model, we entered age, number of science 
courses taken, sex, grade level, pretest quartile, and minority as Level 1 predictors of mean posttest sci¬ 
ence achievement. We observed statistically significant overall effects of sex, pretest quartile, and minor¬ 
ity on mean posttest science achievement {ps < .01). Additionally, we observed that mean posttest sci¬ 
ence achievement for female White students in the lowest quartile on pretest science achievement was 
13.77. Controlling for the effects of all other predictors, every unit increase in quartile on pretest science 
achievement corresponded to a 3.97 unit increase in mean posttest science achievement, being male was 
associated with a 1.67 unit increase in mean posttest science achievement, and minority status was associ¬ 
ated with a 1.50 unit increase in mean posttest science achievement. 
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Table 5 

Student and Classroom Level Predictors of Mean Posttest Science Achievement 


Parameter 

Null Model 

Level 1 Model 

Level 2 Model 

Cohen’s d 

Student level predictors 
of mean science 
achievement: 

Intercept (y 00 ) 

21.99(.54)*** 

13.77(.86)*** 

14.77(1.11)*** 


Age (yio) 

- 

-,55(.34) 

-,64(.37) 


Science Courses (y 20 ) 

- 

.25(.39) 

.30(48) 


Sex (y 30 ) 

- 

1.69(.49)** 

1.34(.60)* 

.21 

Grade Level (y 4 o) 

- 

.77(47) 

.72(.56) 


Pretest Quartile (y 50 ) 

- 

3.97(.25)*** 

3.50(.28)*** 

.54 

Minority (y 60 ) 

- 

1.50(.68)* 

2.31(.77)** 

.36 

Classroom level pre¬ 
dictors of mean sci¬ 
ence achievement 
(i.e., y 0 o): 

Class mean pretest 



.70(.25)** 

.11 

(Yoi) 

Group (y 02 ) 

. 

. 

-,92(.84) 


Classroom level pre¬ 
dictors of 

pretest quartile slope 
(i-e., y 5 o): 

Class mean pretest (y 51 ) 



-.032(44)* 

-.005 

Group (y 52 ) 

- 

- 

.97(.50)* 

.15 

Variance Components: 
Var. in intercept (x 0 o) 

5.46(2.34)*** 

1.66(1.29) 



Age (in) 

- 

.38(.62) 

- 


Higher Science (x 22 ) 

- 

2.08(1.44) 

- 


Sex (X 33 ) 

- 

1.30(1.14) 

- 


Grade Level (x 44 ) 

Pretest Quartile (x 55 ) 

. 

.57(.75) 

.68(.82) 

” 


Minority (X66) 

- 

2.91(1.71) 

- 


a 2 

36.37(6.03) 

16.44(4.05) 

- 



Note. Standard errors for parameters are in parentheses. Standard deviations for variance components 
are in parentheses. *p < .05; **p < .01 ***; p < .001. 


The Level 1 model fit the data substantially 
better than the null model, as evidenced by sub¬ 
stantially lower Akaike Information Criterion 
(AIC) and Bayesian Information Criterion (BIC) 
values (see Table 6). The AIC and BIC are indi¬ 
ces commonly used to assess the relative degree 
to which statistical models, particularly nested 
models, fit the data (Kline, 2010; Raudenbush & 
Bryk, 2002). Reductions in BIC values greater 
than 10 have been suggested as very strong evi¬ 


dence of superior model fit (Raftery, 1995). In 
addition, there was a statistically significant re¬ 
duction in deviance {p < .001). Deviance is an¬ 
other statistic that is commonly used to compare 
the fit of nested models in HLM and can be test¬ 
ed for statistical significance (Raudenbush & 
Bryk, 2002). The Level 1 model explained a 
substantial amount of the between groups (i.e., 
30%) and within groups (i.e., 45%) variance in 
the null model (see Table 6). The variance 
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component for the Level 1 model intercept was Though the between groups variance in the out- 

not statistically significant, suggesting that be- come had been explained, we tested for Level 2 

tween groups there was insubstantial variance effects to assess the effects of the intervention, 

remaining in mean posttest science achievement. 

Table 6 


Proportions of the Variance Explained and Model Fit 


Parameter 

Null Model 

Level 1 Model 

Level 2 Model 

Between group variance explained in 

- 

30% 

- 

intercept 

Within groups variance explained in 

- 

45% 

- 

intercept 

Deviance 

2362.85[3] 

2088.17[36]*** 

1574.13[15] 

AIC (n = 363) 

2368.85 

2160.17 

1604.13 

BIC (n = 363) 

2380.53 

2300.37 

1662.55 


Note. Numbers of parameters are in brackets. *p < .05 **; p < .01 ***; p < .001. 


HLM Level 2 Model 

We specified and tested a model that includ¬ 
ed Level 2 predictors to assess the degree to 
which characteristics of classrooms explained 
the variance in mean posttest science achieve¬ 
ment. In this model, we controlled for the stu¬ 
dent level predictors so that we could assess the 
effects of the intervention on the outcome, over 
and above the effects of the student level predic¬ 
tors. Because the variance components were 
observed to be non-significant for the Level 1 
model, in the Level 2 model we fixed the effects 
for all of the Level 1 predictors. As noted, this 
suggested that between groups there was insub¬ 
stantial variance remaining for these predictors. 
We entered two Level 2 variables (i.e., class 
mean pretest score and treatment group) as pre¬ 
dictors of both mean posttest science achieve¬ 
ment and the effect of pretest quartile on mean 
posttest science achievement to assess the ef¬ 
fects of our intervention. We theorized that the 
effect of these variables on the outcome might 
vary across levels of student pretest science 
achievement: That is, pretest achievement might 
moderate the effects of mean classroom 
achievement and also the treatment on posttest 
science achievement. 

The pretest quartile had a statistically signif¬ 
icant effect on mean posttest science achieve¬ 
ment (p < .001) as in the Level 1 model. Con¬ 


trolling for all other predictors, each unit in¬ 
crease above the lowest quartile on pretest sci¬ 
ence achievement corresponded to a 3.50 unit 
increase in mean posttest science achievement. 
The class mean pretest score was observed to 
have a statistically significant effect on mean 
posttest science achievement {p < .01; control¬ 
ling for all other predictors). Holding all other 
predictors constant, treatment group membership 
was observed to have a statistically significant 
effect on the pretest quartile slope {p < .05). 

For participants in the treatment group class¬ 
rooms, controlling for all other predictors, each 
unit increase above the first pretest quartile cor¬ 
responded to a .97 unit increase in the effect of 
pretest quartile on mean posttest science 
achievement {p < .05). Thus, the effects of the 
treatment were incongruent across levels of pre¬ 
test science achievement (i.e., pretest achieve¬ 
ment moderated the treatment effect). We ob¬ 
served that treatment group membership ap¬ 
peared to have no effect on mean posttest sci¬ 
ence achievement for those in the first quartile 
on pretest science achievement, but a substantial 
positive effect for those in the second, third, and 
fourth quartiles. The magnitude of this effect 
increased for each quartile increase in pretest 
science achievement. See Table 7 for predicted 
scores tabled by condition and pretest science 
achievement quartile, and see Figure 2 for a 
graphical display of these scores. 
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Compared with the Level 1 model, we ob¬ 
served that the Level 2 model BIC was more 
than 80 units smaller and interpreted this as very 
strong evidence of superior fit (Raftery, 1995). 
We also observed a substantial decrease in the 
AIC. Despite the superior AIC and BIC values, 


the deviance value was not observed to change 
with statistical significance. Based on compari¬ 
son of the AIC, BIC, and deviance values for the 
Level 1 and 2 models, we concluded that the 
Level 2 model appeared to fit the data substan¬ 
tially better than the Level 1 model. 


Table 7 

Predicted Posttest Mean Science Achievement by Qucirtile and Experimented Condition 


Pretest Quartile 

Control Group 

Treatment Group 

1 

14.77 

14.77 

2 

18.27 

19.24 

3 

21.77 

23.71 

4 

25.27 

28.18 


Note. Controlling for age, science courses taken, sex, grade level, and pretest quartile at the student 
level. Controlling for class pretest mean at the classroom level. 


HLM 6 produces unstandardized estimates 
of effect size in the metric of the outcome meas¬ 
ure, in this case the TerraNova science test. Be¬ 
cause researchers often want to interpret the 
relative size of effects based on standardized 
estimates, we computed Cohen’s d, an estimate 
of effect size in units of standard deviations 
(Valentine & Cooper, 2003). A d value of .40 is 
generally considered to be a moderate effect size 
in the social sciences, while .80 is considered to 
be a large effect (Cohen, 1988). According to 


Cohen (1988), these effects may be considered 
larger in the field of educational research, where 
effects are often small. The magnitude of the 
intervention’s effects may also be assessed 
through observation of the predicted posttest 
means for prototypical participants, and the per¬ 
cent improvements in the students’ mean post¬ 
test scores. These statistics are reported by quar¬ 
tile and treatment condition in Tables 7 and 8, 
respectively. 


Table 8 


Percentage Change in Number Correct at Posttest (Compared to Controls) 


Pretest Quartile 

Treatment 

1 

+0.0% 

2 

+2.5% 

3 

+4.9% 

4 

+7.3% 


Conclusions and Discussion 

The results of our HLM model analyses in¬ 
dicated that the intervention had a statistically 
significant positive impact on posttest science 
achievement for students in the second, third, 
and fourth quartiles on pretest science achieve¬ 
ment (see Figure 2). Cohen’s d was observed at 
.15 for those in the second quartile, .30 for those 


in the third quartile, and .45 for those in the 
fourth quartile. We interpreted the interven¬ 
tion’s effects as small for participants in the sec¬ 
ond quartile and as moderate for those in the 
third and fourth quartiles. As Cohen (1988) not¬ 
ed, standardized effects are often smaller in edu¬ 
cation compared to those observed in other 
fields. Given the educational nature of this 
study, we judged a d of .30 to be of moderate 
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size, although this would be considered small in 
other fields of research. Relative to the control 
group, the intervention appeared to have no im¬ 
pact on students in the lowest quartile on pretest 
science achievement. The negligible result for 


students in the first quartile seemed counterintui¬ 
tive in relation to the upward trend in the higher 
quartiles, but could not be explained using the 
data available for analysis. 



Figure 2. Predicted posttest mean science achievement by pretest quartile and condition. 


One possible explanation for this result may 
lie in unknown variables associated with the dif¬ 
ference in classrooms between the treatment and 
control groups. It could be the case that stu¬ 
dents’ attitudes toward science in their CTE 
classes contributed to this result; in the focus 
groups, teachers did report variability in re¬ 
sponses from their students. 

Fidelity of treatment data revealed a rate of 
implementation higher than anticipated, given 
the limits of the semester-length implementa¬ 
tion. Fourteen of the fifteen agricultural teach¬ 
ers taught all 15 lessons; 97% (219 of 225) of 
the scheduled lessons were implemented as 
scheduled. In the post-study focus groups, how¬ 
ever, some teachers reported challenges in mov¬ 


ing from Element 4 to Element 5 within the les¬ 
sons, suggesting that the transfer of knowledge 
steps embedded in the pedagogic framework 
may not have fully been articulated in some of 
the lessons. It is possible this impacted the 
achievement of students performing in the lower 
quartile who may have struggled with abstrac¬ 
tions or found it more challenging to make con¬ 
ceptual connections on their own. 

Overall, the findings presented in this study 
support a contextualized learning approach. The 
findings are consistent with those of earlier stud¬ 
ies conducted by Roegge and Russell (1990) and 
Conroy and Walker (2000). As in the Math-in- 
CTE study (Stone et al., 2008), the Science-in- 
CTE model reflects theoretical work in the areas 
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of situated cognition and experiential learning 
(Brown et al., 1989; Lave & Wenger, 1991; 
Paige & Daley, 2009). Specifically, the seven- 
element pedagogical framework is congruent 
with the work of Kali et al. (2008) supporting 
the use of metacognitive approaches in science 
education. 

Additional research should be conducted 
with revisions to the pedagogical model to in¬ 
clude the increased use of inquiry which is con¬ 
sistent with contemporary science teaching 
methods and is an integral part of the national 
science educational standards (National Re¬ 
search Council, 2000). The infusion of inquiry 
into the overall model would be consistent with 
the “crucial ingredients” recommended by 
Bybee (2000, p. 30) for the successful imple¬ 
mentation of inquiry in the science classroom. 

A similar study should be carried out with 
other CTE occupational areas so that generaliza¬ 
tions across different content areas may be 
drawn. Teachers who participated in this inves¬ 


tigation volunteered to participate and were ran¬ 
domly assigned to groups; therefore, an element 
of self-selection existed. Further, teachers also 
received monetary compensation for their partic¬ 
ipation. Would results differ for instructors who 
represented a wider array of teaching abilities, 
levels of motivation, and CTE contexts? 

The results of our analyses indicate that, rel¬ 
ative to the control group, the intervention had a 
statistically significant positive impact on post¬ 
test science achievement for students in the sec¬ 
ond, third, and fourth quartiles on pretest science 
achievement. This supports further investigation 
of the integration of science in CTE classrooms. 
Future studies may seek to increase the amount 
of inquiry in the pedagogical model and apply 
the model to other CTE areas. The pilot Sci- 
ence-in-CTE study also supports continued in¬ 
vestigation of a contextualized approach to in¬ 
fuse science in CTE classrooms as a means of 
furthering the mandate of Perkins IV to integrate 
CTE and academic subjects. 
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